Weighted Ensemble Self-Supervised Learning

Ensembling has proven to be a powerful technique for boosting model performance, uncertainty estimation, and robustness in supervised learning. Advances in self-supervised learning (SSL) enable leveraging large unlabeled corpora for state-of-the-art few-shot and supervised learning performance. In this paper, we explore how ensemble methods can improve recent SSL techniques by developing a framework that permits data-dependent weighted cross-entropy losses. We refrain from ensembling the representation backbone; this choice yields an efficient ensemble method that incurs a small training cost and requires no architectural changes or computational overhead to downstream evaluation. The effectiveness of our method is demonstrated with two state-of-the-art SSL methods, DINO (Caron et al., 2021) and MSN (Assran et al., 2022). Our method outperforms both in multiple evaluation metrics on ImageNet-1K, particularly in the few-shot setting. We explore several weighting schemes and find that those which increase the diversity of ensemble heads lead to better downstream evaluation results. Thorough experiments yield improved prior art baselines which our method still surpasses; e.g., our overall improvement with MSN ViT-B/16 is 3.9 p.p. for 1-shot learning.Comment: Accepted by ICLR 202

Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding

Latent variable models have been successfully applied in lossless compression with the bits-back coding algorithm. However, bits-back suffers from an increase in the bitrate equal to the KL divergence between the approximate posterior and the true posterior. In this paper, we show how to remove this gap asymptotically by deriving bits-back coding algorithms from tighter variational bounds. The key idea is to exploit extended space representations of Monte Carlo estimators of the marginal likelihood. Naively applied, our schemes would require more initial bits than the standard bits-back coder, but we show how to drastically reduce this additional cost with couplings in the latent space. When parallel architectures can be exploited, our coders can achieve better rates than bits-back with little additional cost. We demonstrate improved lossless compression rates in a variety of settings, especially in out-of-distribution or sequential data compression

Moderate Treadmill Exercise Alleviates NAFLD by Regulating the Biogenesis and Autophagy of Lipid Droplet

Lipid droplet is a dynamic organelle that undergoes periods of biogenesis and degradation under environmental stimuli. The excessive accumulation of lipid droplets is the major characteristic of non-alcoholic fatty liver disease (NAFLD). Moderate aerobic exercise is a powerful intervention protecting against the progress of NAFLD. However, its impact on lipid droplet dynamics remains ambiguous. Mice were fed with 15 weeks of high-fat diet in order to induce NAFLD. Meanwhile, the mice performed 15 weeks of treadmill exercise. Our results showed that 15 weeks of regular moderate treadmill exercise alleviated obesity, insulin intolerance, hyperlipidemia, and hyperglycemia induced by HFD. Importantly, exercise improved histological phenotypes of NAFLD, including hepatic steatosis, inflammation, and locular ballooning, as well as prevented liver fat deposition and liver injury induced by HFD. Exercise reduced hepatic lipid droplet size, and moreover, it reduced PLIN2 protein level and increased PLIN3 protein level in the liver of HFD mice. Interestingly, our results showed that exercise did not significantly affect the gene expressions of DGAT1, DGAT2, or SEIPIN, which were involved in TG synthesis. However, it did reduce the expressions of FITM2, CIDEA, and FSP27, which were major involved in lipid droplet growth and budding, and lipid droplet expansion. In addition, exercise reduced ATGL protein level in HFD mice, and regulated lipophagy-related markers, including increasing ATG5, LAMP1, LAMP2, LAL, and CTSD, decreasing LC3II/I and p62, and promoting colocalization of LAMP1 with LDs. In summary, our data suggested that 15 weeks of moderate treadmill exercise was beneficial for regulating liver lipid droplet dynamics in HFD mice by inhibiting abnormal lipid droplets expansion and enhancing clearance of lipid droplets by lysosomes during the lipophagic process, which might provide highly flexible turnover for lipid mobilization and metabolism. Abbreviations: β-actin: actin beta; ATG5: autophagy related 5; LAMP2: lysosomal-associated membrane protein 2; LAMP1: lysosomal-associated membrane protein 1; SQSTM1/p62: sequestosome 1; GAPDH: glyceraldehyde-3-phosphate dehydrogenase; MAP1LC3/LC3: microtubule associated protein 1 light chain 3; ATGL: adipose triglyceride lipase; CSTD: cathepsin D; LAL: lysosomal acid lipase; DGAT1: diacylglycerol-o-acyltransferase 1; DGAT2: diacylglycerol-o-acyltransferase 2; CIDEA: cell death inducing dffa-like effector a; CIDEC/FSP27: cell death inducing dffa-like effector c; FITM2: fat storage-inducing transmembrane protein 2; PLIN2: adipose differentiation related protein; PLN3: tail-interacting protein 47; HSP90: heat shock protein 90; SREBP1c: sterol regulatory element binding protein-1c; chREBP: carbohydrate response element binding protein